Statistical credibility metric for online question answerers

ABSTRACT

Techniques for statistically estimating a rating or other “figure of merit” for a user are disclosed. According to one such technique, a first quantity of submissions that were submitted by a user is determined. A second quantity of submissions that (a) were submitted by the user and (b) obtained a particular rating from a rating mechanism also is determined. A user rating for the user is determined based at least in part on the first quantity, the second quantity, and a factor that is independent of both the first quantity and the second quantity—such as the probability that an answer submitted by any answerer in a population will obtain the particular rating from the rating mechanism. The influence that the second quantity has on the user rating relative to the influence that the factor has on the user rating may depend at least in part on the first quantity.

FIELD OF THE INVENTION

The present invention relates to statistics and, more specifically, to atechnique for determining, statistically, a credibility metric foronline question answerers.

BACKGROUND

Yahoo! Answers is an example of an Internet-accessible answer submissionsystem that allows users all over the world to submit questions thatother users all over the world can view and answer. Users of answersubmission systems submit such questions and answers using an InternetBrowser such as Mozilla Firefox. After a user (an “asker”) has submitteda question, other users can read the question and, if they choose,submit an answer to the question. Answer submission systems typicallyallow users to see, along with a question, answers that have beensubmitted for that question, and the pseudonyms of the users (the“answerers”) who submitted those answers.

A particular question might receive multiple different answers frommultiple different users. Some of the answers might be better thanothers. Answer submission systems may provide a mechanism for users tojudge the value of an answer. For example, Yahoo! Answers provides amechanism whereby the asker can select the best answer submitted for theasker's question. The selected best answer is designated as such. Otherusers can see which of the answers was selected as the best answer.

Over time, some answerers might tend to submit better answers toquestions than other answerers do. This may be the result of theexpertise of some answerers in comparison to the relative inexperienceof other answerers on a particular topic, for example. Answerers whosubmit better answers might tend to have a greater proportion of theiranswers selected by askers as best answers. Consequently, the number ofan answerer's answers that have been selected by askers as best answerscan be used as an indicator of the expertise, wisdom, trustworthiness,and/or reliability of that answerer. At least one answer submissionsystem provides a mechanism whereby everyone can see the number of bestanswers that have been submitted by each answerer. Askers might givemore weight to answers provided by a particular answerer if a highpercentage of the particular answerer's answers have been selected asbest answers.

For each question, though, there can only be one best answer selectedfrom among potentially many answers submitted for that question.Consequently, the vast majority of answers submitted for each questionwill not be selected as best answers. Unless they are extraordinary,most answerers do not have a very good chance of having many of theiranswers selected as best answers. However, with each answer that aparticular answerer submits, the chances that at least one of theparticular answerer's answers will be selected as a best answer improveat least marginally. As a result, answerers who have submitted a largequantity of answers tend to have more reliable “best-answer indicators”than answerers who have submitted fewer answers.

The traditional best-answer indicator, which is sometimes defined as theproportion of an answerer's answers that have been selected as bestanswers, sometimes fails to reflect accurately the true merit andcredibility of that answerer. An answerer who has submitted only oneanswer probably will not have his answer selected as a best answer, withthe result that his best-answer indicator will have a value of zero. Asa result, many new answerers will end up with best-answer indicatorsthat seem to indicate that those answerers' answers are not highlycredible—even though, in reality, those answerers might be verycredible. Those new answerers might go on to have a greater-than-averageproportion of their answers selected as best answers over time.Unfortunately, when an answerer has submitted answers to only a fewquestions, the number of that answerer's answers that have been selectedas best answers is often not a very accurate reflection of thatanswerer's merit or credibility. Alternatively, in the unusual instancein which an answerer has only submitted a single answer and that answerhas been chosen as a best answer, the value of that answerer'sbest-answer indicator will be one, indicating that his answers arealways selected as best answers—which is virtually impossible.

The approaches described in this section are approaches that could bepursued, but not necessarily approaches that have been previouslyconceived or pursued. Therefore, unless otherwise indicated, it shouldnot be assumed that any of the approaches described in this sectionqualify as prior art merely by virtue of their inclusion in thissection.

DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings and in whichlike reference numerals refer to similar elements and in which:

FIG. 1 is a flow diagram that illustrates an example of a technique forestimating a credibility rating for an answerer, according to anembodiment of the invention; and

FIG. 2 is a block diagram of a computer system on which embodiments ofthe invention may be implemented.

DETAILED DESCRIPTION

In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be apparent, however,that the present invention may be practiced without these specificdetails. In other instances, well-known structures and devices are shownin block diagram form in order to avoid unnecessarily obscuring thepresent invention.

Overview

According to one embodiment of the invention, a credibility rating or“figure of merit” for an answerer is determined using a statisticaltechnique that does not unfairly penalize or unfairly reward theanswerer due to sampling error associated with having answered only afew questions.

According to this technique, the answerer's “true” credibility rating isapproximated by statistically estimating the probability that theanswerer's future answer will be selected as the best answer to aquestion. Although the answerer's “true” credibility rating—the actualprobability that the answerer's future answer will be selected as a bestanswer—cannot be known with certainty in most cases, the answerer's“true” credibility rating can be estimated statistically. In order toavoid unfairly penalizing or unfairly rewarding an answerer who hasanswered only a few questions, the answerers' estimated credibilityrating is determined based at least in part not only on the answerer'sown actual previous performance, but also on the “composite” previousperformance of a population of other answerers of which the answerer isa part.

At first, when the answerer has answered only a few questions, theproportion of the answerer's answers that have been selected as bestanswers usually will be a poor indicator, on its own, of the answer's“true” credibility rating; given a very small sample of the answerer'sanswers, the proportion of those answers that have been selected as bestanswers is, statistically, not very determinative of the probabilitythat a future answer of that specific answerer will be selected as abest answer. Therefore, in one embodiment of the invention, the extentto which the answerer's previous performance influences the answerer'scredibility rating is based on the number of answers that the answererhas submitted. If the answerer has submitted relatively few answers,then the answerer's previous performance (i.e., the proportion of theanswerer's answers that have been selected as best answers) influencesthe answerer's credibility rating to only a relatively minor extent.Alternatively, if the answerer has submitted relatively many answers,then the answerer's previous performance influences the answerer'scredibility rating to a relatively major extent. As the number ofanswers submitted by the answerer becomes sufficiently large, theinfluence that the answerer's previous performance has on the answerer'scredibility rating approaches complete and exclusive influence.

Later, when the answerer has answered very many questions, theproportion of the answerer's answers that have been selected as bestanswers is an excellent indicator, on its own, of the answer's “true”credibility rating; given a very large sample of the answerer's answers,the proportion of those answers that have been selected as best answersis, statistically, very determinative of the probability that a futureanswer of that specific answerer will be selected as a best answer.

In one embodiment of the invention, the answerer's credibility rating isinfluenced, to varying extents, by the previous performance of apopulation of other answerers of which the answerer is a member. Givensuch a population of answerers, a “composite” previous performance ofthose answerers as a group may be determined statistically. When verylittle or nothing can be ascertained about an answerer's “true”credibility due to the lack of actual previous performance dataavailable for the answerer, it is reasonable to assume that theanswerer's predicted future performance will be at least somewhatsimilar to the predicted future performance of a theoretical answererwho represents the statistical composite of all answerers in thepopulation of which the answerer is a member. Therefore, in oneembodiment of the invention, as the number of answers submitted by ananswerer approaches zero, the influence that the population's previousperformance has on the answerer's credibility rating approaches completeand exclusive influence. Conversely, as the number of answers submittedby an answerer approaches infinity, the influence that the population'sprevious performance has on the answerer's credibility rating wanes tono influence whatsoever. As the amount of actual previous performancedata for the answerer grows, the answerer's actual previous performancedata supplants, more and more, the population's previous performancedata in estimating the answerer's credibility rating.

Techniques for statistically estimating an answerer's credibility ratingor “figure of merit” are described in greater detail below.

Estimating an Answerer's Credibility Rating

According to one embodiment of the invention, an answerer's credibilityrating is characterized by the probability that the answerer's answerwill be selected as a best answer for a question in a category or set ofquestions being considered. This probability is represented by θ.Because θ is considered to be a hidden variable, θ can only be estimatedstatistically from specific observations associated with the answerer.Such observations include the number of the answerer's answers that havebeen selected as best answers, n_(B), and the number of answers that theanswerer has submitted, n.

Embodiments of the invention address the potentially high degree ofuncertainty that is associated with estimating a credibility rating whenthe estimated credibility rating is based on a small number of theanswerer's own answers.

According to one embodiment of the invention, an answerer's credibilityrating is estimated based on a “Bayesian smoothing” technique, which isdescribed below. In one embodiment of the invention, the credibilityrating is estimated based on a mixture of overall population statisticsand statistics of the specific answerer. The greater the number ofanswers that the answerer has submitted, the greater will be thecontribution of the answerer's specific statistics to the answerer'scredibility rating relative to the contribution of the statistics of theoverall population.

According to one embodiment of the invention, at the extremes, for n=0,the only statistics available for the answerer are those of the overallpopulation. Therefore, when n=0, the statistics of the overallpopulation will dominate the estimation of the answerer's credibilityrating for small n. For large n(n→∞), however, the estimation of theanswerer's credibility rating will be completely dominated by thespecific answerer's statistics, making the overall population statisticsirrelevant.

Discussed below are details of an example of one technique forestimating a specific answerer's credibility rating based on both hisown prior performance and the prior performances of a population ofusers. In one embodiment of the invention, a two-dimensional contingencytable h(n,n_(B)) contains the counts (numbers) of answerers who have, todate, answered n questions and had n_(B) of those answers rated as bestanswers by the question askers. An observed distribution, as captured byh(n,n_(B)), is combined with the particular n and n_(B) values for asingle specific answerer in order to obtain a “safe” (in the sense ofneither over penalizing nor over rewarding the single answerer due tosampling error inherent in his particular observed n and n_(B) values)estimates (computed ratings). There are many different ways in whichsuch a combination might be made in order to obtain such estimates.

For example, the following estimate might be used:

${\theta^{*} = {{\lbrack {1 - {\beta (n)}} \rbrack ( \frac{n_{B}}{n} )} + {{\beta (n)}\overset{\_}{\theta}}}},$

Where θ is the probability of any answer given by any answerer in aspecified population being selected as a best answer. The function β(n)may be some simple monotonically decreasing function of n such ase^(−λn), where λ is a positive adjustable parameter.

As an estimate of this, the total number of best answers divided by thetotal number of answers may be used. This may be written (and computed)as:

$\overset{\_}{\theta} = {\frac{\sum\limits_{n = 0}^{n_{MAX}}{\sum\limits_{n_{B} = 0}^{n}{n_{B}{h( {n_{B},n} )}}}}{\sum\limits_{n = 0}^{n_{MAX}}{h{\sum\limits_{n_{B} = 0}^{n}( {n_{B},n} )}}}.}$

As another potential definition for θ, the average value of the ratio

$( \frac{n_{B}}{n} )$

over the population of answerers might be computed. That is:

$\overset{\_}{\theta} = {\frac{\sum\limits_{n = 0}^{n_{MAX}}{\sum\limits_{n_{B} = 0}^{n}{( \frac{n_{B}}{n} ){h( {n_{B},n} )}}}}{\sum\limits_{n = 0}^{n_{MAX}}{\sum\limits_{n_{B} = 0}^{n}{h( {n_{B},n} )}}}.}$

Another approach follows the Bayesian paradigm for statisticalestimation, a key feature of which is the prior distribution π(θ), whichis estimated from h(n,n_(B)). One estimate of this is:

${{\pi (\theta)} = \frac{\sum\limits_{\{{n,{{n_{B}{({n_{B}/n})}} = \theta}}\}}^{\;}{h( {n,n_{B}} )}}{\sum\limits_{n = 0}^{n_{MAX}}{\sum\limits_{n_{B} = 0}^{n}{h( {n,n_{B}} )}}}},$

where the sum in the numerator is over all pairs (n,n_(B)) for which

$( \frac{n_{B}}{n} ) = {\theta.}$

Unfortunately, this is a discrete distribution that only has non-zerovalues for a finite enumerable set of θ values. One way to address thisis to fit the distribution h(n,n_(B)) to a smooth/continuous probabilitydistribution. This distribution may be a specific parametric functionalform like the Beta distribution or, if the distribution is notwell-described by a single Beta distribution, a mixture (weighted sum)of Beta distributions can be used. Beta distributions (includingmixtures of Betas) are appealing because they are conjugate in Bayesianterminology. An implication of this is that integrals of the products ofbinomial distributions and Beta distributions, such as those used incertain embodiments of the invention, evaluate to closed forms, whichare computationally convenient. There are a large number of mathematicalfitting and smoothing techniques that may be applied to obtain a smoothprior distribution π(θ) from the observations tabulated in {h(n,n_(B))}.

Alternative Contexts and General Applicability

Although techniques described herein are described in the context ofdetermining a credibility rating for an answerer, techniques describedherein also may be applied to other contexts. When applied to othercontexts, some aspects of the techniques described herein may differ.For example, the fundamental quantity that is being used as a measuremay change. In some embodiments of the invention described herein, thisfundamental quantity is the probability θ, but some other fundamentalquantity might be selected when techniques described herein aregeneralized or adapted to other contexts.

For another example, the functional form of the probability distributionthat relates directly observable quantities (e.g., n_(B) given n) to thefundamental quantity (e.g., θ) might differ. In one embodiment of theinvention, the probability distribution is a binomial distribution thatis described in further detail below, and the functional form is adirect theoretical consequence of assuming that that answerer'sperformance is describable by the parameter θ, which is the probabilitythat an answerer's answer will be selected as a best answer. However,when techniques described herein are applied to other problems, theremay be some latitude in choosing the probability distribution (thefunctional form of the distribution) to be used.

For another example, the functional form of the prior distribution(e.g., π(θ)) of the fundamental quantity might differ. Although in oneembodiment of the invention the functional form of the priordistribution is a mixture of Beta distributions described in furtherdetail below, in other embodiments of the invention, the functional formof the prior distribution might take a different form.

The Posterior Distribution

According to one embodiment of the invention, an answerer's credibilityrating is estimated based on the following idealization. An answerer isassumed to have an associated attribute θ, which is the probability thatan answer given by the answerer will be chosen by the question asker asa best answer. In one embodiment of the invention, θ is assumed to beindependent of other factors such as the category of the question andthe identity of the asker. In such a scenario, if the answerer answers nquestions, the probability that n_(B) of those answers will be chosen asbest answers is given by the binomial distribution:

$\begin{matrix}{{p( {{n_{B}n},\theta} )} = {\begin{pmatrix}n \\n_{B}\end{pmatrix}{\theta^{n_{B}}( {1 - \theta} )}^{n - n_{B}}}} & (1)\end{matrix}$

The answerer's credibility rating or “figure of merit” may be any one ofa number of possible estimates of θ or measures that are statisticallyassociated with θ. According to one embodiment of the invention, inorder to obtain fairer estimates of the answerer's credibility rating, aBayesian statistical paradigm is adopted. In this paradigm, a priordistribution π(θ) is assumed. Based on the prior distribution π(θ), aposterior distribution p(θ|n_(B),n) may be formed, where:

$\begin{matrix}{{p( {{\theta n_{B}};n} )} = \frac{{p( {{n_{B}\theta};n} )}{\pi (\theta)}}{p( {n_{B};n} )}} & (2)\end{matrix}$

One way of constructing the prior distribution π(θ) in the foregoingequation is by using the (n,n_(B)) data for all answerers in apopulation of answerers (e.g., all Yahoo! Answers users). Many differenttechniques might be used to construct the prior distribution π(θ). Anexample technique for constructing the prior distribution π(θ) isdescribed in greater detail below.

Constructing the Prior Distribution

In one embodiment of the invention, a prior distribution π(θ) isconstructed based on data collected for multiple answerers in apopulation of answerers (e.g., the n and n_(B) values for each answererin the population). In order to obtain a closed form for the priordistribution π(θ), the Beta distribution may be used as the priordistribution. Alternatively, a mixture of Beta distributions might befitted together to form a prior distribution. The Beta distribution isconjugate to the Binomial distribution, and therefore gives a closedform when integrated with the Binomial distribution.

The Beta distribution, generally, is given by the following equation:

$\begin{matrix}{{f( {{x;\alpha},\beta} )} = \frac{{x^{\alpha - 1}( {1 - x} )}^{\beta - 1}}{\int_{0}^{1}{{u^{\alpha - 1}( {1 - u} )}^{\beta - 1}{u}}}} \\{= {\frac{\Gamma ( {\alpha + \beta} )}{{\Gamma (\alpha)}{\Gamma (\beta)}}{x^{\alpha - 1}( {1 - x} )}^{\beta - 1}}} \\{= {\frac{1}{B( {\alpha,\beta} )}{x^{\alpha - 1}( {1 - x} )}^{\beta - 1}}}\end{matrix}$

where α and β are parameters that must be greater than zero, Γ( ) is theGamma function, and B( ) is the Beta function.

Using a single Beta distribution, the prior distribution π(θ) is then:

$\begin{matrix}{{\pi ( {{\theta;\alpha},\beta} )} = {\frac{1}{B( {\alpha,\beta} )}{\theta^{\alpha - 1}( {1 - \theta} )}^{\beta - 1}}} & (3)\end{matrix}$

and if a mixture of Beta distributions is used, then the priordistribution π(θ) is of the form:

$\begin{matrix}{{{\pi ( {{\theta;\alpha},\beta,\eta} )} = {\sum\limits_{i = 1}^{m}{\frac{\eta_{i}}{B( {\alpha_{i},\beta_{i}} )}{\theta^{\alpha_{i} - 1}( {1 - \theta} )}^{\beta_{i} - 1}}}},} & (4)\end{matrix}$

Where α, β, and η are m-component vectors and η is normalized such that

${\sum\limits_{i = 1}^{n}\eta_{i}} = 1.$

Statistical Values Usable to Determine a Credibility Rating

Several different statistical values that are either derived from orstatistically associated with θ may be used as, or to at least partiallydetermine, an answerer's credibility rating or “figure of merit.” Onesuch value that may be used is the “maximum a posteriori” (MAP) estimate{circumflex over (θ)}_(MAP) of θ, which is given by the followingequation:

{circumflex over (θ)}_(MAP) =arg max p(θ|n _(B) ;n)   (5)

The above formula is for a probability density, which can be integratedover a finite interval to obtain a finite (non-infinitesimal)probability. For example:

$\begin{matrix}{{\gamma = {{p( {{{\theta > \theta_{\gamma}}n_{B}},n} )} \equiv {\int_{\theta_{\gamma}}^{1}{{p( {{\theta n_{B}},n} )}{\theta}}}}},} & (6)\end{matrix}$

where p(θ>θ_(γ)|n_(B),n) is the conditional probability that θ>θ_(γ)given the observed values n_(B) and n. Another value that may be usedas, or to at least partially determine, an answerer's credibility ratingor “figure of merit” is θ_(γ).

The two rating measures {circumflex over (θ)}_(MAP) and θ_(γ) areessentially types of statistical estimates of θ. These estimates areformulated within the Bayesian statistical paradigm. One of theseestimates, θ_(γ) is an interval estimate in which the upper limit of theinterval is θ=1. That is, θ is understood to lie within the interval[θ_(γ),1] with probability γ. This is an instance of the Bayesianconcept of a credible interval (CI), which is the counterpart of aconfidence interval in classical statistics. This type of credibleinterval is referred to as being “one-sided” because one of the limitsof the interval (the upper one in this case) is the end of the range ofpossible values for the parameter (max(θ)=1).

With this understanding, the following relationships may be applied:

${p( {{\theta n_{B}};n} )} = \frac{{p( {{n_{B}\theta};n} )}{\pi (\theta)}}{p( {n_{B};n} )}$${p( {{n_{B}\theta};n} )} = {\begin{pmatrix}n \\n_{B}\end{pmatrix}{\theta^{n_{B}}( {1 - \theta} )}^{n - n_{B}}}$p(n_(B); n) = ∫₀¹p(n_(B)θ; n)π(θ)θ${P( {{{\theta > \theta_{\gamma}}n_{B}},n} )} = \frac{\int_{\theta_{\gamma}}^{1}{{p( {{n_{B}\theta};n} )}{\pi (\theta)}{\theta}}}{\int_{0}^{1}{{p( {{n_{B}\theta};n} )}{\pi (\theta)}{\theta}}}$

Another value that may be used as, or to at least partially determine,an answerer's credibility rating or “figure of merit” is the posteriorexpectation {tilde over (θ)} of θ, which is given by the followingequation:

$\begin{matrix}{\overset{\sim}{\theta} \equiv {E_{\pi}( {{\theta n_{B}},n} )} \equiv {\int_{0}^{1}{\theta \; {p( {{\theta n_{B}},n} )}{\theta}}}} & (7)\end{matrix}$

The fact that the parameter θ is interpreted as a probability means that{tilde over (θ)} is the probability that the next answer that theanswerer gives will be selected as a best answer, given the answerer'shistorical performance data (i.e., (n,n_(B))) and assuming the priordistribution π(θ).

Although the parameter θ is used in the above equations, which can beused to determine credibility ratings for an answerer, in an alternativeembodiment of the invention, similar statistics of a numerical score sgiven to the best answer by the asker may be used instead of using θ.

Example Flow

FIG. 1 is a flow diagram that illustrates an example of a technique forestimating a credibility rating for an answerer, according to anembodiment of the invention. The technique described is merely oneembodiment of the invention. The technique, or portions thereof, may beperformed, for example, by one or more processes executing on a computersystem such as that described below with reference to FIG. 2. Forexample, the technique may be performed by one or more processesexecuting on an Internet-accessible server.

According to one embodiment of the invention, a credibility rating for aparticular answerer is re-determined or recomputed whenever theparticular answerer answers another question and/or whenever one of theparticular answerer's answers is selected as a best answer. Such actionstypically will modify the n and/or n_(B) values maintained for theparticular user. The particular answerer's credibility rating may beupdated within a database.

In block 104, a quantity n of answers that were submitted by aparticular answerer (of potentially many answerers who submitted answersto a question) is determined. In one embodiment of the invention, thequantity n is the total number of answers that the particular answererhas submitted to one or more questions. The server may maintainstatistics that indicate, for each answerer, the total number of answersthat the answerer has submitted.

In block 106, a quantity n_(B) of answers that were both (a) submittedby the particular answerer and (b) selected or designated as a “best” or“selected” answer by askers (or other people besides the particularanswerer) is determined. For example, if the particular answerersubmitted 10 answers to 10 different questions, and if 2 of thoseanswers were selected as “best” answers by the askers of thosequestions, then the server may determine that the particular answerer'sn is 10, and that the particular answerer's n_(B) is 2. The server maymaintain statistics that indicate, for each answerer, the total numberof the answerer's answers that have been selected as “best” answers.

Although in one embodiment of the invention the quantity n_(B) indicatesthe quantity of the particular answerer's “best answers,” in alternativeembodiments of the invention, the quantity n_(B) may indicate a quantityof the particular answerer's submissions (whether answers or other typesof submissions) that have obtained at least a specified rating (whetherthe rating is a “best answer” rating, a numerical rating, or some othertype of rating) from some specified rating mechanism (whether themechanism comprises rating by question submitters or some other kind ofrating mechanism).

In block 108, based at least in part on the quantity n and the quantityn_(B), a first probability that a future answer submitted by theparticular answerer will be designated or selected as a “best” or“selected” answer by question askers (or other people besides theparticular answerer) is determined.

In block 10, a second probability is determined. The second probabilityis the probability that an answer, submitted by any answerer in aspecified population of multiple answerers that includes the particularanswerer, will be designated or selected as a “best” or “selected”answer by question askers (or other people besides those who submittedthe answer). For example, the second probability may be determined usingthe Bayesian statistical techniques described above. The secondprobability may be determined based on a prior distribution π(θ) thathas been constructed using the collected n and n_(B) values for all ofthe answerers in the specified population, for example.

Although in one embodiment of the invention the second probability isthe probability that an answer, submitted by any answerer in a specifiedpopulation of multiple answerers that includes the particular answerer,will be designated or selected as a “best” or “selected” answer, inalternative embodiments of the invention, a factor other than the secondprobability may be used instead of the second probability wherever thesecond probability would be used. However, like the second probability,this other factor may be independent of both the particular answerer's nand n_(B).

In block 112, a credibility rating for the particular answerer isdetermined based at least in part on the first probability, the secondprobability, and the quantity n determined for the particular answerer.For example, the credibility rating may be determined based at leastpartially on the statistical estimates {circumflex over (θ)}_(MAP),θ_(γ), and/or {tilde over (θ)} as described above. More specifically, inone embodiment of the invention, the influence that the firstprobability has on the credibility rating, relative to the influencethat the second probability has on the credibility rating, is based onthe quantity n determined for the particular answerer. For example, if nis large, then the first probability may have a major influence on theanswerer's credibility rating while the second probability may have aminor influence on the answerer's credibility rating. In contrast, if nis small, then the first probability may have a minor influence on theanswerer's credibility rating while the second probability may have amajor influence on the answerer's credibility rating. In one embodimentof the invention, as n increases, the influence that the firstprobability has on the answerer's credibility rating increases while theinfluence that the second probability has on the answerer's credibilityrating decreases. The computation of the statistical estimates{circumflex over (θ)}_(MAP), θ_(γ), and {tilde over (θ)} described aboveincorporates the varying influences of the particular answerer's own nand n_(B) values and the collected n and n_(B) values for all of theanswerers in the specified population.

In one embodiment of the invention, the credibility rating is normalizedso that the credibility rating represents a corresponding value within aspecified range of values (e.g., 1 to 10, 0 to 100, etc.). As isdiscussed above, the answerer's credibility rating may be stored in adatabase in association with other information about the answerer.

In block 114, a Hypertext Transfer Protocol (HTTP) request is receivedover a network. For example, a server may receive such a request overthe Internet. Such a request might originate from an Internet browser,such as Mozilla Firefox, executing on a computer that is locatedremotely from the server. Such a request might be a request for adynamically generated web page that indicates answers to a question,pseudonyms of answerers who submitted those answers, and credibilityratings of those answerers, for example.

In block 116, the requested web page is dynamically generated. Forexample, the server may generate the web page dynamically. In oneembodiment of the invention, when generated, the requested web pageindicates the particular answerer's answer to a particular question, theparticular answerer's pseudonym, and the particular answerer'scredibility rating (potentially along with the answers, pseudonyms, andcredibility ratings of other answerers who also submitted answers to theparticular question).

In block 118, a HTTP response is sent over a network toward the originof the HTTP request discussed with reference to block 114. For example,the server may send the HTTP response over the Internet. The HTTPresponse includes data that represents the web page. When an Internetbrowser that executes at the origin of the HTTP request receives theHTTP response, the Internet browser responsively displays the web page,including the particular answerer's credibility rating. Thus, by sendingthe HTTP response, the server essentially causes the particularanswerer's credibility to be displayed.

In one embodiment of the invention, a “composite” answerer rating isdetermined for the specified population of multiple answerers discussedabove. The composite answerer rating is estimated based on the collectedn and n_(B) values for all of the answerers in the specified population.The composite answerer rating reflects the probability that an answersubmitted by any answerer in the specified population will be designatedor selected as a “best” or “selected” answer by question askers (orother people besides those who submitted the answer). In such anembodiment of the invention, the particular answerer's rating may bedetermined based at least in part on this composite answerer rating andthe particular answerer's own n and n_(B) values. The influence that thecomposite answerer rating has on the particular answerer's ratingdecreases as the particular answerer's own n value increases. Incontrast, the influence that the particular answerer's own n_(B) valuehas on the particular answerer's rating increases as the particularanswerer's own n value increases.

Hardware Overview

FIG. 2 is a block diagram that illustrates a computer system 200 uponwhich an embodiment of the invention may be implemented. Computer system200 includes a bus 202 or other communication mechanism forcommunicating information, and a processor 204 coupled with bus 202 forprocessing information. Computer system 200 also includes a main memory206, such as a random access memory (RAM) or other dynamic storagedevice, coupled to bus 202 for storing information and instructions tobe executed by processor 204. Main memory 206 also may be used forstoring temporary variables or other intermediate information duringexecution of instructions to be executed by processor 204. Computersystem 200 further includes a read only memory (ROM) 208 or other staticstorage device coupled to bus 202 for storing static information andinstructions for processor 204. A storage device 210, such as a magneticdisk or optical disk, is provided and coupled to bus 202 for storinginformation and instructions.

Computer system 200 may be coupled via bus 202 to a display 212, such asa cathode ray tube (CRT), for displaying information to a computer user.An input device 214, including alphanumeric and other keys, is coupledto bus 202 for communicating information and command selections toprocessor 204. Another type of user input device is cursor control 216,such as a mouse, a trackball, or cursor direction keys for communicatingdirection information and command selections to processor 204 and forcontrolling cursor movement on display 212. This input device typicallyhas two degrees of freedom in two axes, a first axis (e.g., x) and asecond axis (e.g., y), that allows the device to specify positions in aplane.

The invention is related to the use of computer system 200 forimplementing the techniques described herein. According to oneembodiment of the invention, those techniques are performed by computersystem 200 in response to processor 204 executing one or more sequencesof one or more instructions contained in main memory 206. Suchinstructions may be read into main memory 206 from anothermachine-readable medium, such as storage device 210. Execution of thesequences of instructions contained in main memory 206 causes processor204 to perform the process steps described herein. In alternativeembodiments, hard-wired circuitry may be used in place of or incombination with software instructions to implement the invention. Thus,embodiments of the invention are not limited to any specific combinationof hardware circuitry and software.

The term “machine-readable medium” as used herein refers to any mediumthat participates in providing data that causes a machine to operate ina specific fashion. In an embodiment implemented using computer system200, various machine-readable media are involved, for example, inproviding instructions to processor 204 for execution. Such a medium maytake many forms, including but not limited to, non-volatile media,volatile media, and transmission media. Non-volatile media includes, forexample, optical or magnetic disks, such as storage device 210. Volatilemedia includes dynamic memory, such as main memory 206. Transmissionmedia includes coaxial cables, copper wire and fiber optics, includingthe wires that comprise bus 202. Transmission media can also take theform of acoustic or light waves, such as those generated duringradio-wave and infra-red data communications.

Common forms of machine-readable media include, for example, a floppydisk, a flexible disk, hard disk, magnetic tape, or any other magneticmedium, a CD-ROM, any other optical medium, punchcards, papertape, anyother physical medium with patterns of holes, a RAM, a PROM, and EPROM,a FLASH-EPROM, any other memory chip or cartridge, a carrier wave asdescribed hereinafter, or any other medium from which a computer canread.

Various forms of machine-readable media may be involved in carrying oneor more sequences of one or more instructions to processor 204 forexecution. For example, the instructions may initially be carried on amagnetic disk of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over atelephone line using a modem. A modem local to computer system 200 canreceive the data on the telephone line and use an infra-red transmitterto convert the data to an infra-red signal. An infra-red detector canreceive the data carried in the infra-red signal and appropriatecircuitry can place the data on bus 202. Bus 202 carries the data tomain memory 206, from which processor 204 retrieves and executes theinstructions. The instructions received by main memory 206 mayoptionally be stored on storage device 210 either before or afterexecution by processor 204.

Computer system 200 also includes a communication interface 218 coupledto bus 202. Communication interface 218 provides a two-way datacommunication coupling to a network link 220 that is connected to alocal network 222. For example, communication interface 218 may be anintegrated services digital network (ISDN) card or a modem to provide adata communication connection to a corresponding type of telephone line.As another example, communication interface 218 may be a local areanetwork (LAN) card to provide a data communication connection to acompatible LAN. Wireless links may also be implemented. In any suchimplementation, communication interface 218 sends and receiveselectrical, electromagnetic or optical signals that carry digital datastreams representing various types of information.

Network link 220 typically provides data communication through one ormore networks to other data devices. For example, network link 220 mayprovide a connection through local network 222 to a host computer 224 orto data equipment operated by an Internet Service Provider (ISP) 226.ISP 226 in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the“Internet” 228. Local network 222 and Internet 228 both use electrical,electromagnetic or optical signals that carry digital data streams. Thesignals through the various networks and the signals on network link 220and through communication interface 218, which carry the digital data toand from computer system 200, are exemplary forms of carrier wavestransporting the information.

Computer system 200 can send messages and receive data, includingprogram code, through the network(s), network link 220 and communicationinterface 218. In the Internet example, a server 230 might transmit arequested code for an application program through Internet 228, ISP 226,local network 222 and communication interface 218.

The received code may be executed by processor 204 as it is received,and/or stored in storage device 210, or other non-volatile storage forlater execution. In this manner, computer system 200 may obtainapplication code in the form of a carrier wave.

In the foregoing specification, embodiments of the invention have beendescribed with reference to numerous specific details that may vary fromimplementation to implementation. Thus, the sole and exclusive indicatorof what is the invention, and is intended by the applicants to be theinvention, is the set of claims that issue from this application, in thespecific form in which such claims issue, including any subsequentcorrection. Any definitions expressly set forth herein for termscontained in such claims shall govern the meaning of such terms as usedin the claims. Hence, no limitation, element, property, feature,advantage or attribute that is not expressly recited in a claim shouldlimit the scope of such claim in any way. The specification and drawingsare, accordingly, to be regarded in an illustrative rather than arestrictive sense.

1. A method of rating a particular user, the method comprising:determining a first quantity of submissions that were submitted by theparticular user; determining a second quantity of submissions that (a)were submitted by the particular user and (b) obtained a particularrating from a rating mechanism; determining a composite user rating thatreflects a probability that any submission submitted by any user in apopulation of submitters will obtain the particular rating from therating mechanism; determining a particular user rating for theparticular user based at least in part on the composite user rating, thefirst quantity, and the second quantity; and displaying the particularuser rating.
 2. The method of claim 1, wherein, as the first quantityincreases, an influence that the composite user rating has on theparticular user rating decreases and an influence that the secondquantity has on the particular user rating increases.
 3. A method ofrating a particular user, the method comprising: determining a firstquantity of submissions that were submitted by a particular user;determining a second quantity of submissions that (a) were submitted bythe particular user and (b) obtained a particular rating from a ratingmechanism; determining a particular user rating for the particular userbased at least in part on the first quantity, the second quantity, and afactor that is independent of both the first quantity and the secondquantity; and displaying the particular user rating; wherein, indetermining the particular user rating, an influence that the secondquantity has on the particular user rating relative to an influence thatthe factor has on the particular user rating is based at least in parton the first quantity.
 4. The method of claim 3, wherein, in determiningthe particular user rating, as the first quantity increases, theinfluence that the second quantity has on the particular user ratingincreases and the influence that the factor has on the particular userrating decreases.
 5. A method of rating a particular user, the methodcomprising: determining a first quantity of answers that were submittedby the particular user; determining a second quantity of answers that(a) were submitted by the particular user and (b) designated as selectedanswers by one or more users other than the particular user;determining, based at least in part on the first quantity and the secondquantity, a first probability that an answer submitted by the particularuser will be designated as a selected answer by one or more users otherthan the particular user; determining a second probability that aparticular answer submitted by any user in a specified group of two ormore users will be designated as a selected answer by one or more usersother than a user that submitted the particular answer; determining arating for the particular user based at least in part on the firstprobability, the second probability, and the first quantity; anddisplaying the rating as a measure of credibility of answers submittedby the particular user; wherein, in determining the rating, an influencethat the first probability has on the rating relative to an influencethat the second probability has on the rating is based at least in parton the first quantity.
 6. The method of claim 5, wherein, in determiningthe rating, as the first quantity increases, the influence that thefirst probability has on the particular user rating increases and theinfluence that the second probability has on the rating decreases.
 7. Amethod of rating a particular user, the method comprising: forming aprior distribution based on statistics compiled for a population ofmultiple users; based on both the prior distribution and statistics thatare specific to the particular user, estimating a rating for theparticular user.
 8. The method of claim 7, wherein, in estimating therating, as a quantity of submissions submitted by the user increases,the influence that the particular user-specific statistics have on therating increases and the influence that the prior distribution has onthe rating decreases.
 9. The method of claim 7, wherein forming theprior distribution comprises constructing the prior distribution basedon collected data about submissions that previously were submitted byusers in the population of users.
 10. The method of claim 7, whereinestimating the rating comprises estimating the rating based at least inpart on at least one of: (a) a maximum a posteriori estimate of aspecified probability, (b) a credible interval estimate of the specifiedprobability, and (c) a posterior expectation of the specifiedprobability.
 11. A computer-readable medium that stores instructionswhich, when executed by one or more processors, cause the one or moreprocessors to perform the method of claim
 1. 12. A computer-readablemedium that stores instructions which, when executed by one or moreprocessors, cause the one or more processors to perform the method ofclaim
 2. 13. A computer-readable medium that stores instructions which,when executed by one or more processors, cause the one or moreprocessors to perform the method of claim
 3. 14. A computer-readablemedium that stores instructions which, when executed by one or moreprocessors, cause the one or more processors to perform the method ofclaim
 4. 15. A computer-readable medium that stores instructions which,when executed by one or more processors, cause the one or moreprocessors to perform the method of claim
 5. 16. A computer-readablemedium that stores instructions which, when executed by one or moreprocessors, cause the one or more processors to perform the method ofclaim
 6. 17. A computer-readable medium that stores instructions which,when executed by one or more processors, cause the one or moreprocessors to perform the method of claim
 7. 18. A computer-readablemedium that stores instructions which, when executed by one or moreprocessors, cause the one or more processors to perform the method ofclaim
 8. 19. A computer-readable medium that stores instructions which,when executed by one or more processors, cause the one or moreprocessors to perform the method of claim
 9. 20. A computer-readablemedium that stores instructions which, when executed by one or moreprocessors, cause the one or more processors to perform the method ofclaim 10.